Acoustic and Lexical Modeling Techniques for Accented Speech Recognition

Speech interfaces are becoming pervasive among the common public with the prevalence of smart phones and cloud-based computing. This pushes Automatic Speech Recognition (ASR) systems to handle wide range of environments including different channels, noise conditions and speakers with varying accents. This thesis focuses on the impact of speakers’ accents on the ASR models and techniques to make them robust to such variations. State-of-the-art large vocabulary ASRs perform poorly when presented with accented speech, that is either unseen or under-represented in the training data. Current approaches to handle accent variations mainly involve adaptation of acoustic models or the pronunciation dictionary. This thesis examines novel adaptation algorithms capable of modeling changes in phonological realizations, that uniquely characterize accent variations. Techniques that can exploit the contemporary availability of extensive, albeit unlabeled data resources are also investigated. We design experiments under various scenarios where accent adaptation is critical for speech recognition. In target accent adaptation setup, a source ASR trained on resourcerich accent(s) is adapted to a target accent with limited adaptation data. We propose semi-continuous decision tree adaptation and multi-gram pronunciation models to efficiently model the pronunciation changes between source and target accents. Active and semi-supervised learning are studied to extend the improvements obtained from supervised adaptation. We introduce relevance criteria based data selection to sample additional accent-specific data from large, unlabeled speech corpora with multiple accents. Finally, we generalize the target accent adaptation techniques to handle multiple accents in the training set. We formulate an accent adaptive training framework using factorized models with shared canonical parameters and accent-specific modules. Our proposed algorithms will be evaluated on Arabic and English accents and compared against existing adaptation techniques.